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ABSTRACT 

The Hierarchical SEGmentation (HSEG) algorithm, which 
is a combination of hierarchical step-wise optimization and 
spectral clustering, has given good performances for hyper- 
spectral image analysis. This technique produces at its out- 
put a hierarchical set of image segmentations. The automated 
selection of a single segmentation level is often necessary. 
We propose and investigate the use of automatically selected 
markers for this purpose. In this paper, a novel Marker-based 
HSEG (M-HSEG) method for spectral-spatial classification 
of hyperspectral images is proposed. First, pixelwise classi- 
fication is performed and the most reliably classified pixels 
are selected as markers, with the corresponding class labels. 
Then, a novel constrained marker-based HSEG algorithm is 
applied, resulting in a spectral-spatial classification map. The 
experimental results show that the proposed approach yields 
accurate segmentation and classification maps, and thus is at- 
tractive for hyperspectral image analysis. 

Index Terms — Hyperspectral images, hierarchical seg- 
mentation, classification, marker selection. 

1. INTRODUCTION 

Hyperspectral imaging records a detailed spectrum of light 
arriving in each pixel [1], which, makes it. possible to iden- 
tify physical materials and classify regions within the im- 
age scene with much higher accuracies when compared to 
panchromatic or multispectra! sensors. While most classifi- 
cation techniques process each pixel independently using its 
spectral values only (one of the most frequently used tech- 
niques are Support Vector Machines (SVM) [2]), recent stud- 
ies have shown the advantage of including information about 
spatial dependencies for accurate image analysis, i.e., per- 
forming spectral -spatial classification [3, 4], 

In previous works, we have distinguished spatial struc- 
tures in the image by performing unsupervised segmentation. 
Then, each region was classified by applying a majority vote 
rule over the pixelwise classification results [5J. The Hierar- 
chical SEGmentation (HSEG) method, which is a combina- 


tion of hierarchical step-wise optimization and spectral clus- 
tering f6J, has shown good performance for spatial analysis of 
hyperspectral images. Unlike most other segmentation tech- 
niques, the HSEG generates at its output a hierarchical set of 
image segmentations. It is often necessary to choose one or 
several relevant hierarchical segmentation levels. The auto- 
mated selection of the hierarchical level(s) can be achieved by 
incorporation of some additional knowledge into a segmenta- 
tion procedure. In this paper, we propose and investigate the 
use of automatically derived markers , or region seeds, for this 
purpose. 

In [4). we have proposed to use probability estimates ob- 
tained by the pixelwise SVM classification in order to choose 
the most reliably classified pixels as markers of spatial re- 
gions. Furthermore, a Minimum Spanning Forest (MSF) 
rooted on the selected markers was constructed, resulting in a 
segmentation and classification map. The described approach 
has performed well in classification. 

In this paper, wc adapt the classification-based approach 
for marker selection proposed in [4], in order to define rele- 
vant markers for the HSEG procedure. Thus, a new Marker- 
based HSEG (M-HSEG) method for spectral-spatial classifi- 
cation of hyperspectral data is proposed. First, probabilistic 
pixelwise classification is applied and a map of markers is 
constructed by selecting the most reliably classified pixels. 
Then, a novel constrained M-HSEG algorithm is applied, re- 
sulting in a classification map. We propose and discuss sev- 
eral ways of integrating markers into the HSEG technique. 

The paper is organized as follows. In the next section, 
a new M-HSEG approach is presented. Experimental results 
are discussed in Section 3. Finally, conclusions are drawn in 
Section 4. 

2. PROPOSED CLASSIFICATION METHOD 

The flow-chart of the proposed M-HSEG classification method 
is shown in Fig, 1 , A hyperspectral image can be considered 
as a set of n pixel vectors X = {xj e j = 1, 2, .... n\. 
Classification consists in assigning each pixel to one of K in- 



formation classes. In the following, each step of the proposed 
procedure is described. 

2.1. Classification-based marker selection 

As we proposed in [4], markers of spatial regions are auto- 
matically selected using classification results. The following 
algorithm is applied for this purpose: 

1) Pixelwise classification: A probabilistic pixelwise 
SVM classification of a hyperspectral image is performed [2, 

7], This step results in a classification map (where each pixel 
has a unique class label) and a probability map (containing 
probability estimates for each pixel to belong to the assigned 
class). 

2) Marker selection: A connected components labeling 
is applied on the classification map. Then, each connected 
component is analyzed as follows: 

• If a region is large (number of pixels in the regions > 
M), its marker is defined as the P% of pixels within 
this region with the highest probability estimates. 

• If a region is small, its potential marker is formed by the 
pixels with probability estimates higher than a defined 
threshold S. 

At the output of the marker selection step, a map of rn 
markers is given, where each marker O, = {xj 6 X, j — 

1. 2. carci(Oj); Lo i }(i = 1, ....to) consists of one or sev- 
eral pixels (which are not necessarily spatially connected) and 
has a class label £o, ■ 

2.2. Marker-based HSEG 

The following outline of the HSEG algorithm is based on the 
description given in [6]: 

1) Initialize the segmentation by assigning a region la- 
bel for each pixel. If a pre-segmentation is provided, label 
each pixel according to the pre-segmentation. Otherwise, la- 
bel each pixel as a separate region. 

2) Calculate the Dissimilarity Criterion (DC) value be- 
tween all pairs of spatially adjacent regions. 

3) Find the smallest DC value dissirnj'al and set thre.sh-.vai 
equal to it. Then merge all pairs of spatially adjacent regions 
with dissim-val — thresluval. 

4) If the parameter S v , g ht > 0.0, merge all pairs of spa- 
tially non-adjacent regions with 

dissim.val < S W ght • fhreshjval. 

5) If convergence is not achieved, go to step (2). 

Different measures can be used for computing DCs be- 
tween regions, such as LI norm, Infinity (Inf) norm, Spectral 
Angle Mapper (SAM) between the region mean vectors [6], 
The optional parameter S wg ht tunes the relative importance 
of spectral clustering versus region growing. 

The main idea behind the marker-based HSEG algorithm 
consists in assigning a marker label for each region containing 



Fig. 1. Flow-chart of the proposed M-HSEG classification 
scheme. 

marker pixels, and merging regions with an additional con- 
dition: two regions with different marker labels can not be 
merged together. The proposed M-HSEG algorithm can be 
summarized as follows: 

• Initialize the segmentation by labeling either the whole 
marker, or a separate non-marker pixel as one region. 
Assign for every region the corresponding marker label 
(which is equal to zero for non-marked regions). 

• At. each iteration, perform HSEG, with an additional 
condition: two regions with different non-zero marker 
labels have the DC equal to infinity (in practice, the 
upper maximum value of float) and are never merged 
together. When a marked region is merged with a non- 
marked region, the resulting region keeps the marker 
label inherited from the marked region. 

• Stop the iterative process when either a number of re- 
gions is equal to the number of markers (thus, no more 
merging is possible) or the smallest DC between any 
two neighboring regions is higher than the preset (or 
computed) threshold. 

2.3. Implementations of the marker-based HSEG algo- 
rithm 

We have investigated the performance of three different im- 
plementations of the proposed M-HSEG approach. One im- 
plementation, M-HSEG 1 ", is based on the description given in 
the previous subsection. All the pixels belonging to the same 
marker are initialized as one region, and then iterative region 
merging is performed. 

A second implementation, M-HSEG 71 , first initializes 
each pixel as one region and assigns a marker label for every 
region (equal to zero for non-marked regions). During the 
region merging procedure, the regions with equal non-zero 




Table 1 . Information Classes, Number of Labeled Samples and Classification Accuracies in Percentage for the Center of Pavia 
Image: Overall Accuracy (OA), Average Accuracy (AA), Kappa Coefficient (k) and Class-Specific Accuracies. 
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marker labels have a zero DC value, while the regions with 
different non-zero marker labels have the DC equal to infinity. 

A third implementation, M-HSEG op , first initializes each 
pixel as one region. If the given pixel is marked, the corre- 
sponding region obtains a new non-zero marker label, with 
the corresponding information class. Thus, at the initializa- 
tion step all the markers are split into one-pixel markers. 
Then, iterative region merging is performed, providing that 
regions with different markers cannot be merged together. At 
the final step, the regions containing pixels of the same initial 
marker are merged together. 

The implementations M-HSEG P and M-HSEG op can be 
useful when images contain large regions with high intra- 
regiors spectral variation. In this case, it may be advantageous 
to compute region feature vectors over parts of these regions. 

3. EXPERIMENTAL RESULTS AND DISCUSSION 

We applied the proposed M-HSEG approach to two hyper- 
spectral airborne images described in the following; 

1 ) The Center of Pavia image was acquired by the ROSTS 
sensor over the urban area of Pavia, Italy. The image is of 785 
by 300 pixels, with a spatial resolution of 1.3 m/pixel and 102 
spectral channels. A three-band false color image is shown in 
Fig, 2. Nine classes of interest are considered, which are de- 
tailed in Table 1, with the number of labeled samples for each 
class. Thirty samples for each class were randomly chosen 
from the reference data as training samples. The remaining 
samples composed the test set. 

2) The Indian Pines image is of vegetation area that was 
recorded by the AVIRIS sensor. It is of 145 by 145 pixels, 
with a spatial resolution of 20 m/pixel, 200 spectral channels 
and sixteen information classes. More information about the 
image, with the used training-test set can be found in [4]. 

For both images, the probabilistic one-versus-one SVM 



Fig. 2. Center of Pavia image, (a) Three-band color compos- 
ite. (b) M-HSEG" 1 ' classification map (SAM DC, S wg ht = 
0.2). (c) SVMMSF classification map (SAM DC). 


Table 2. Global Classification Accuracies in Percentage for 
the Indian Pines Image: Overall Accuracy (OA), Average Ac- 
curacy (AA), and Kappa Coefficient ( k ). 
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classification with the Gaussian radial basis function kernel 
was applied. The optimal parameters C (penalty during the 
SVM optimization) and 7 (spread of the RRF kernel) were 
chosen by fivefold cross-validation. Then, marker selection 
was performed with parameters M = 20, P = 40%. The 
threshold S was chosen to be equal to the lowest probabil- 
ity within the highest 2 % of the probability estimates for the 
whole image [4], 

Finally, the M-HSEG segmentation of the images was per- 
formed, using the three proposed implementations. The LI 
norm, the Inf norm and the SAM between the region mean 
vectors were applied as DCs. In all experiments, the M-HSEG 
algorithm has been run until no more merging was possible. 
By assigning the class of each marker to the region containing 
this marker, the classification maps were obtained. 

Table 1 summarizes global (overall, average accura- 
cies and kappa coefficient [5J) and class-specific accura- 
cies of the pixelwise SVM classification and the proposed 
method for the Center of Pavia image, using the LI norm 
and the SAM between the region mean vectors as DCs and 
Swght = [0.0, 0.2, 0.5}. In order to compare the results 
of the proposed method with other advanced classification 
techniques, we have included results obtained using the con- 
struction of an MSF from the same set of markers (SVMMSF 
method) [4], Table 2 reports global accuracies of the SVM 
and SVMMSF classification and the proposed method for the 
Indian Pines image, using the SAM and the Inf norm between 
the region mean vectors as DCs and S WfJ ht — [0.0, 0.2] . The 
following conclusions can be drawn: 

• The proposed marker-based M-HSEG method yields 
accurate segmentation and classification results. The 
average accuracy is improved by 3.1 and 7.3 percent- 
age points when compared to the SVM classification, 
for the Center of Pavia and the Indian Pines images, 
respectively. Therefore, it is useful to include markers 
in the HSEG algorithm, in order to automatically select 
the relevant segmentation level. 

• The M-HSEG" P implementation significantly outper- 
forms M-HSEG' 1 " and M-HSEG P implementations in 
terms of accuracies. Thus, a region mean vector seems 
to be a “poor ” representative feature of image regions. 

• The M-HSEG h p method with S V!g ht =0,2 performs 
in most cases better than when S wg t !t = 0.0 is used. 
However, classifications accuracies decrease with fur- 
ther increase of the S wg ht value. 

• The M HSEG' 7 ' classification results are in most 
cases non-significantly lower when compared to the 
SVMMSF results. The reason may lie in the fact that 
region mean vectors do not well represent enough im- 
age regions. However, ail the DCs give similar results 
when applying the M-HSEG approach. The SVMMSF 
method gives significantly lower accuracies when the 
SAM DC is applied for classification of (he Center of 
Pavia image. It assigns large portions of the water to 


the spatially adjacent asphalt regions, and assimilates 
shadows with neighboring regions (see Fig. 2). Thus, 
the M-HSEG technique appears to be more robust 
when using different DCs. 

4. CONCLUSIONS 

A new marker-based HSEG method for spectral-spatial clas- 
sification of hyperspectral images was presented in this pa- 
per. In this method, a marker map is first constructed using 
probabilistic classification results. Then, the novel M-HSEG 
algorithm is applied, resulting in a spectral-spatial classifica- 
tion map. Several ways of integrating markers into the HSEG 
technique are proposed and investigated. Experimental re- 
sults demonstrate that the proposed method yields accurate 
classification maps and is sufficiently robust for classifying 
different kinds of images. In the future, we plan to explore 
the choice of optima! representative features for segmentation 
regions, in order to further improve segmentation and classi- 
fication results. 
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